Repository navigation
Conversation
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the
|
|
@codecov-ai-reviewer review |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
EvalRunner.run()initialized every report as passed and returnedpassed=Truewhen anEvalSuitecontained no query, replay, or agent cases. The evaluation API and promoter only checked that a suite result existed, so a suite with no executable evaluation could be treated as green.This is a vacuous evaluation result. It weakens the intended fail-closed evaluation gate, although this change does not claim a separately verified exploitable security vulnerability.
Reproduction
An
EvalSuitewith emptyqueries,replays, andagentsproduced:{"cases": [], "passed": true}The same occurred when only the currently unimplemented
coverageconfiguration was present.Solution
no executable eval cases.The change does not alter the
EvalSuitemodel or add new DSL validation rules. Suites containing executable cases retain their existing behavior.Tests
uv run --directory ontogeny --group dev pytest tests/test_evolve.py -q(63 passed)uv run ruff check ontogeny/server/ontogeny/evolve/evalrunner.py ontogeny/tests/test_evolve.pyuv run ruff format --check ontogeny/server/ontogeny/evolve/evalrunner.py ontogeny/tests/test_evolve.pyBase branch:
ontogeny